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ABSTRACT 



The benefits of and issues associated with output-related 
funding (ORF) were assessed by examining the use of ORF in vocational 
education and training (VET) in the European Union and the United States. 

Data were gathered in the following ways: several online bibliographic 
searches; consultation with 54 experts, including VET researchers, national 
policymakers, and lobbyists; CEDEFOP requesjts for information from 
organizations and individuals; and traditional library searches. ORF was 
defined as basing funding on outputs produced, which are generally measured 
in terms of the achievement of qualifications for school-based training 
and/or job attainment. The emphasis given to ORF in the VET programs examined 
ranged from 75% (the United Kingdom's Training for Work program) to 5% (Job 
Training Partnership Act programs) . ORF was determined to offer the following 
benefits: gives training providers more flexibility in the type of provision 
offered; enhances improvements in performance; increases value for money by 
providing incentive to fulfill certain achievements and discouraging 
"time-serving" in training, which does not lead to outcomes; and simplifies 
administration and clarifies audit requirements. ORF ' s success in achieving 
efficiency, reducing administrative costs, and enhancing accountability could 
not be determined unequivocally because it was rarely used as the sole 
instrument of performance management. (57 references) (MN) 
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This Report aims to inform policy-makers about the issues arising from and practice of 
funding Vocational Education and Training (VET) on the basis of programme outcomes 
rather than enrolment/attendance (i.e. switching the emphasis of funding from inputs to 
outputs). It gathers together in a single dossier an outline of the policy issues output- 
related funding (ORF) generates as well as a number of real-world examples of its use in 
countries within and beyond the European Union. The objective of the Report, 
therefore, is to provide a useful summary document of evidence on ORF and its 
practice, which policy-makers and others may wish to consult. 

ORF offers several benefits to policy-makers. These include: 

• giving training providers more flexibility in the type of provision offered 

• enhancing improvements in performance 

• increasing value for money by providing an incentive to fulfil certain achievements 
and by discouraging ‘time-serving’ in training which does not lead to outcomes 

• simplifying administration and clarifying audit requirements 

Set against these benefits are a number of operational issues which need to be tackled 
before ORF can be introduced, as well as a number of consequences and implications 
which can arise from its implementation and use. These can be summarised in a series 
of policy questions: 

• What level of ORF is required to reap the optimum benefits of tying funding to 
output? 

• Is ORF passed on to all those in the contracting chain? 

• How is output defined? 

• How is output measured? 

• How are funding uncertainties minimised? 

• Do the measures of performance used to financially reward and penalise training 
providers act as good proxies for the long-term labour market prospects of trainees? 

• Are only the best candidates ‘creamed off onto output-related funded programmes? 

• To what extent are eligibility criteria used to counter the danger of ‘creaming’ by 
requiring training providers to ‘dredge’? 

• How is participation of disadvantaged members of the labour market protected? 
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What consequences does ORF have for the content and occupational spread of 
training provision? 

Does ORF lead to cash flow problems among training providers? 

Are particular types of training provider more at risk than others from the operation 
of ORF? 

Does ORF raise barriers to entry for new training providers? 

Are the administrative rules and regulations that accompany ORF simpler or more 
complicated than other systems? 

Does ORF reward the value added by training intervention? 

Does ORF reward all or only some of the value added by training providers? 

What checks and balances are in place to ensure probity and reliability? 

What mechanisms are used to counter these problems? 

How successful is ORF in achieving its aims? 



The Report outlines the operation of ORF in countries within and beyond the EU with 
the above policy questions in mind. In particular, attention is focused on the British 
system of funding government training schemes, the US system of training the 
disadvantaged and poor through a federally funded programme, and Dutch proposals to 
use some form of ORF in the near future to finance adult and vocational training. As 
such the Report provides a wealth of material for policy-makers and other interested 
parties to reflect on a funding arrangement from which other EU Member States may 
wish to borrow or learn. 
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1.1 Aims and Objectives 



This Report aims to inform policy-makers about the issues arising from and practice of 
funding VET on the basis of programme outcomes rather than enrolment/attendance (i.e. 
switching the emphasis from inputs to outputs). It outlines in a single dossier the policy 
issues output-related funding (ORF) generates as well as a number of real-world 
examples of its use in countries within and beyond the European Union (EU). The 
objective of the Report, therefore, is to provide a useful summary document of evidence 
on ORF and its practice, which policy-makers and others may wish to consult. 

It should be noted at the outset that ORF in the Report is discussed in the context of 
public sector provision of initial vocational training and training for the unemployed - 
although the discussion could also apply to the public provision of continuing vocational 
training. The terms ‘output’ and ‘outcome’ are used interchangeably throughout the 
Report. 



1.2 Methods 



Material for the Report was gathered in a number of ways; four avenues in particular 
were pursued. First, several on-line bibliographic searches were launched using 
different search engines and a range of keywords. The latter included ‘output-related 
funding’, ‘performance-related funding’, ‘financing training’, ‘funding training’ and 
‘creaming’ as well as titles of specific training programmes. Through these means, the 
contents pages of recent (post- 1990) social science journals were searched. If the title 
and abstract seemed relevant to the focus of the project, the articles were consulted. 

Secondly, in order to ensure that the material available for inclusion in the Report was 
as up-to-date and comprehensive as possible, international Vocational and Education 



Training (VET) experts were contacted by letter, fax or email. They were asked to 
supply information or contacts relevant to the Report’s focus. A total of 54 experts, 
including VET researchers, national policy-makers, lobbyists and representatives of 
international organisations, were contacted in this way during the life of the project. 
Most of these were based in the EU, and nearly all of them responded with information 
of one sort or another. 

Thirdly, CEDEFOP carried out a number of searches of its own and lodged requests for 
information with organisations and individuals with whom it is in regular contact, 
making additional documents available to the project. 

The fourth material-gathering device was the traditional library search. Two libraries in 
particular were the focus of attention: the University of Leicester library and the 
Department for Education and Employment (DfEE) library in Sheffield. Keyword 
searches were made at both. 



1.3 Outline of the Report 



The Report consists of two parts: 

• an outline of some of the economic and social issues arising from ORF regimes 

• three examples of ORF usage from both within and outside the European Union 

The Report is structured accordingly. Section Two discusses the policy issues which 
may arise. Section Three provides information on how ORF has so far been used to fund 
VET in several countries and Section Four concludes the Report with a short summary 
which brings the two major parts of the Report together. 
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This section of the Report discusses what output-related funding (ORF) is, the rationale 
and principles behind its introduction, a series of operational issues, and a number of 
consequences and implications which can arise from its implementation and use. 
Within these sub-sections a number of issues associated with output-related funding will 
be discussed. At the end of the section, the discussion will be summarised by posing a 
series of policy questions to be considered when evaluating and assessing ORF regimes. 

In this section, the discussion takes place in the abstract and makes no judgements about 
the very different VET funding systems across the EU. The points made below are 
general issues associated with output-related funding, which could vary according to 
individual VET systems and the policies and programmes which govern them. 



2.1 What is Output-Related Funding? 

There are three main activities or events to which the funding of vocational education 
and training (VET) can be related: 

• enrolment (sometimes called ‘starts’ or ‘entries’) 

• the duration and nature of programmes (i.e. course length, attendance requirements 
and infrastructure needs) 

• the outputs produced (usually measured in terms of qualifications achievement for 
school-based training and/or job attainment with regard to labour market training) 

Output-related funding (ORF) refers to the last of these mechanisms. It can be used 
singularly or in combination with the other two. 

All three mechanisms are related in some way to the performance of training 
institutions, measured in terms of enrolment, attendance or attainment. Hence, the use 
of the term performance-related funding can sometimes lead to confusion. Indeed, 



several experts consulted for this project conceived performance-related funding in such 
an all-encompassing way. This Report, therefore, avoids using this term wherever 
possible. Having said this, it is important to note that the term is commonly used in the 
US literature (see Section Three below) to refer to what this report calls ORF. 

For this Report, output-related funding is defined as funding which is linked in some 
way to a pre-defined ‘successful’ and ‘measurable’ output of the VET system. What 
might constitute a ‘successful’ and ‘measurable’ output to the training system is 
discussed in more detail below. 

It is important to point out that an ORF mechanism is unlikely to be introduced as a 
single measure; rather, it will be the result of other reforms and changes to the VET 
structure and funding regime. As it operates on the supply side, it will usually be 
introduced as part of a package of measures aimed at improving the delivery of training 
by decentralising the management of programmes and resources to those nearer the 
point of delivery. As a result, ORF is often used as a means of ensuring that those 
charged with administering government programmes (whether private or public sector 
organisations) are financially accountable for the success, or otherwise, of the outcomes 
delivered. In other words, ORF is often used as a means of monitoring performance and 
ensuring accountability. 



2.2 What is its Rationale? 



In general terms, the rationale behind introducing output-related funding is to create an 
incentive-based funding mechanism which aims to improve (or discourage) certain pre- 
defined outputs as a means to reward, or penalise, training institutions. Within this 
general principle there are more specific ones which depend on individual VET systems 
and the relationships between the various stakeholders. 

Nevertheless, the arguments in favour of ORF have widespread applicability, which may 
appeal to those concerned with ‘training’ issues as well as to those concerned with 
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‘funding’. It is worth rehearsing these in some detail at this point, so that policy-makers 
can balance the potential benefits against the possible costs ORF may bring. 



2.2.1 ORF as an incentive to improve efficiency 

By making a proportion of a training institution’s budget dependent on the achievement 
of a particular output(s), an incentive is immediately created for institutions to improve 
or change certain aspects of their training policies in order to increase their chance of 
generating the desired output(s). Depending on the definition of output, this could 
include focusing on the quality of the training delivered and/or the assessment and 

guidance offered to trainees prior to, or after, training. By re-focusing and re-assessing 

certain aspects of the training process, the efficiency of the whole training system may 
be enhanced. For example, more emphasis could be placed on placing trainees in 
appropriate courses where they will have a reasonable chance of success, closely 

monitoring their progress, and taking quick corrective action when a trainee is 

underachieving or exhibits an inclination to drop out. Furthermore, provided the output- 
related funding regime is correctly weighted to maximise achievement and support 
training for those with special needs, ORF can offer training providers financial 
inducements to take on trainees with special needs and provide an incentive to stretch all 
trainees to their fullest potential. 

In addition, funding on the basis of training weeks can encourage stakeholders to think 
in terms of time-serving, and can be an incentive to training providers to retain trainees 
for an unnecessarily long time. Paying for outcomes, on the other hand, gives an 
incentive to reduce training time to the point necessary for trainees to achieve the 
‘output’, whether this is concerned with gaining a qualification and/or obtaining 
employment. It also means that training provider income is geared to success. If a 
trainee leaves earlier than expected, but gains an outcome, the training provider’s 
income is not affected. Conversely, lengthy training times are not rewarded. 
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2.2.2 ORF as a means of enhancing flexibility 



Linking funding to outcomes could provide an incentive for training providers to offer a 
broader and more flexible range of training options. Providers could opt for a long, 
low-cost-per-week solution, or a short, high-cost-per-week arrangement, with the same 
overall financial effects; the ability to do this will depend on existing regulations 
regarding the time spent in training. Nevertheless, the flexibility inherent in an ORF 
model might be of benefit to all concerned: to trainees whose training can be more 
closely tailored to their needs; to training providers designing programmes; and to 
programme administrators, who can better match training and labour market needs and 
trainees’ aspirations. 



2.2.3 ORF as a means of reducing administrative costs and bureaucracy 

Detailed attendance records on all participants are required if funding is tied to the time 
spent in training. This means that considerable resources are devoted to the collection, 
monitoring, and auditing of attendance records; such resources might be better spent on 
delivering the training itself. A process-based funding system, therefore, might be 
costly to maintain, complex, and open to abuse. One of the potential benefits of ORF is 
that it can be used to simplify administrative systems, help to develop clearer audit 
paths, and hence enhance provider accountability. The volume of data to be checked 
under an outcomes regime is potentially more manageable and the documentary 
evidence required is for discernible one-off events. This can release resources to be 
used for training delivery. 



2.2.4 ORF as a means of improving transparency and accountability 

Last but not least, ORF provides a financial incentive to fulfil certain achievements and 
therefore provides a mechanism for policy-makers to emphasise the delivery of the 
outputs they consider essential. This could mean adding to the overall stock of 
qualifications held by the workforce or getting people into jobs. Tying funding directly 
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to these policy goals can make these training objectives and their achievement more 
transparent, and therefore makes the parties involved more accountable. 



2.3 Operational Issues 

The introduction of ORF requires that a number of operational issues be dealt with 
before the potential benefits outlined above can be realised. Again, these issues will 
largely depend on the way in which individual funding and training systems are 
structured. However, some general points can be made. 



2.3.1 ORF mechanisms 

Although the principle of ORF is straightforward, it is rarely used in its purest form, i.e. 
funding for outputs only, for a number of reasons outlined further below. Instead, ORF 
is used in combination with enrolment- and attendance-based funding; these are often 
referred to as ‘starts and outcomes’ and ‘weeks and outcomes’ regimes. In both cases 
funding is split into two parts. Under the ‘starts and outcomes’ regime, one part of the 
payment is triggered by each trainee who enrols in the programme, while the other is 
due for each trainee who exits the programme with a particular outcome. The ‘weeks 
and outcomes’ regime maintains a stronger link with the process of training, where 
payment is based on trainee attendance as well as trainee performance on completion. 

Under either regime the emphasis given to ORF can vary. Indeed, as Section Three 
shows, the UK gives significantly more emphasis to ORF than the US. Moreover, the 
tendency in the UK has been to tilt the balance even further in the direction of ORF in 
order to maximise the potential benefits it brings. This raises the question of the 
optimal balance between input-related and output-related funding that maximises the 
potential benefits of ORF (some of these have been outlined above), while minimising 
its associated costs (these are outlined further below). 
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2.3.2 Institutional structure 



It should be noted that the funding chain between government and training provider is 
often mediated by other organisations. This means that ORF contracts between 
government and training intermediaries need not be passed on in full, or even in part, to 
training providers. ORF, therefore, needs to be studied at various points along the 
contractual chain, as it cannot be assumed that agreements reached at one level are 
automatically passed on unchanged or unmodified to those further down. This point is 
made in more detail in the case studies contained in Section Three. 



2.3.3 Defining a successful output 

A key operational issue is: What is defined as a ‘successful’ output? While this might 
be politically determined, one can single out two major outputs commonly sought from 
government-funded VET. First, the acquisition of skills as shown by qualification 
attainment; this can refer to the grade/level achieved, the number of certificates awarded 
and/or the area of study. Second, output may be defined, in both initial training and 
training for the unemployed, simply as getting trainees into jobs. 



2.3.4 Measuring a successful output 

Associated with this operational issue is how outputs, once defined, are measured. In 
principle, there are three ways of doing so: ‘absolute’, ‘relative’ and ‘value-added’ 
measures. The ‘absolute’ measure takes no account whatsoever of the quality of intake; 
that is to say, institutions are measured according to the outputs they achieve, regardless 
of the quality of the recruits they are able to attract. The second, ‘relative’ measure is 
designed to reward enhanced provider performance by linking funding to improvements 
training providers are able to make to their past performance. However, neither an 
absolute nor a relative measure links inputs to outputs. The third measure - ‘value- 
added’ - attempts to achieve this feat, thereby isolating the contribution training 
providers alone make to output. This can be achieved by comparing ‘the characteristics 
and attainment of learners at entry (the input data) and their achievements at exit (the 
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output data)’ (Further Education Unit, 1993, quoted in Further Education Development 
Agency, 1995: 1). In practice, this means finding an indicator which acts as an 
approximation for every training participant’s ‘input’ value into the training process and 
establishing whether the process of training leads to higher ‘output’ value, i.e. the value 
added to the stock of human capital. Such a measure, therefore, aims to identify the 
expected output of the training process, given that there are differences in input levels. 
This ensures that the achievement of a ‘successful’ output is within the ability of every 
training institution, since success is measured by the difference between inputs and 
outputs. 



2.3.5 Funding an ORF regime 

Another key operational issue is how the regime is to be funded. One of the central 
advantages Of the system of ORF is that it encourages training providers to achieve 
increasingly better outputs. However, this raises difficulties from a funding perspective, 
since public expenditure commitments will also rise (fall) in line with increased 
(decreased) training provider performance. One way of countering funding uncertainty 
is to place a ceiling on the payments that can be made to training providers, thereby 
capping funding at a certain level. However, there is a trade-off between giving training 
providers greater financial incentives to do well and knowing with some certainty what 
is likely to be spent. Inevitably, a compromise between these two positions has to be 
reached before ORF can become operational. 



2.4 Some Consequences and Implications of ORF 

Like other funding mechanisms, ORF can have a number of consequences when put into 
practice. Many of these are revealed by the evaluations reviewed in Section Three - 
some have led to modifications of the ORF regime, others have even prompted 
institutional reform. It is, therefore, worth considering, albeit briefly, the possible 
consequences and implications of ORF in relation to its potential benefits in the abstract 
before considering them in real-world situations in Section Three. Again, these 
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consequences and implications will depend on the existing policies and regulations 
concerning individual training and funding structures. 



2.4.1 ORF: efficiency and equity? 

• Measuring short-term performance 

The system of ORF usually rests on observed outcomes shortly after the completion of a 
training programme. This is partly due to the nature of recurrent funding and the need 
to minimise the time-lag between training delivery and the receipt of payment (i.e. cash 
flow). Definitions of successful outputs include participants in work at a designated 
week after graduation, their weekly earnings that week or the number of hours worked 
during that week. However, these are short-term indicators of performance and may be 
poor proxies for the evaluation of the long-term benefits of investment in human 
resources. In other words, training programmes may temporarily raise the labour market 
prospects of those who participate, but the advantage they gain may quickly disappear. 
Rewarding training providers for longer-term effects, however, is likely to lead to 
measurement and cash flow difficulties. Measuring output on the basis of short-term 
indicators might affect the long-term efficiency of the funding and training system. 

• ‘Creaming’ and ‘dredging’ 

Operators may react to ORF by enrolling only those most likely to meet the output 
criteria, i.e. they will ‘cream’. This is more probable in cases where the applicant pool 
is much larger than the available places and hence the scope for selection is enhanced. 
Similarly, under conditions of limited resources, the tendency is to ‘cream’. On the 
other hand, this issue is less of a problem where the applicant pool is small and 
resources are more readily available. This means that when there is a large and diverse 
pool of eligible participants, screening by training providers can be undertaken to 
identify promising candidates, while weeding out those who might be more 
troublesome. Those with special needs are the most likely to be excluded. 

In economic literature this is referred to as the problem of ‘moral hazard’ or ‘the folly of 
rewarding A, while hoping for B’. Moral hazard occurs when A and B diverge. This is 



O 

ERIC 



- 14 - 



19 



a common problem in subcontracting arrangements which require the subcontractor to 
provide a service rather than a tangible product to another group of individuals. The 
provision of government-funded training is one such example. If the goal is to enhance 
someone’s employment chances, then the output indicator needs to be able to measure 
this effect of the training, rather than encouraging training providers to screen for those 
with better prospects which would exist regardless of the training. A value-added 
measure of output, as well as other institutional reforms, may be used to counter 
creaming. 

On the other hand, programme objectives may be set to extend rather than limit training 
opportunities to the most disadvantaged in society. In these circumstances, there is no 
‘floor’ to the selection criteria; however, there is a ‘ceiling’ beyond which participation 
is not permitted. The ‘ceiling’ could be defined by education levels, income or previous 
employment. As a result, only the most disadvantaged form the available pool from 
which selection takes place. This practice is termed ‘dredging’ and is usually justified 
for reasons of equity. In so far as government training schemes have eligibility criteria, 
dredging, by definition, can be said to operate. However, the narrower the eligibility 
criteria, the greater the extent of dredging required by policy-makers. Moreover, the 
criteria can be changed to refocus training activity on those most in need, thereby 
minimising the extent of creaming. 

• Altering patterns of participation 

As a result of creaming, it is often alleged that ORF alters the pattern of participation 
and shifts provision away from the most disadvantaged groups in society. Notable 
among the ‘losers’ are women, ethnic minorities, the poorly educated and those with 
special needs. However, ORF mechanisms themselves may be designed to counter this 
tendency by, for example, attaching more funds to outputs attained by disadvantaged 
groups, i.e. dredging, as discussed above. 
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2.4.2 ORF: enhancing flexibility? 



• Encouraging/discouraging certain types of training 

In funding systems which take no account of the variable costs of training for different 
occupations, training provision may shift towards ‘low cost’ forms of training such as 
business administration and away from ‘high cost’ areas such as engineering. 
Participants may also be steered away from more difficult programmes and placed in 
courses for which it is easier to demonstrate ‘success’ according to the output criteria. 
This shift may occur regardless of the skills and training demands of employers, and 
therefore the qualifications gained may be inappropriate. ORF regimes, however, may 
be weighted to counter this tendency, with more costly or more ‘necessary’ training 
carrying higher output funding. 

• Changing the content of training 

The content of training may change as a result of ORF depending on the way in which 
‘output’ is defined. For example, where ORF is linked to job placement, then on-the- 
job training and job search assistance may be more effective and quicker in placing 
people into jobs than, for example, classroom-based training. Shifts in the content of 
training are therefore likely. Training times, too, may change. If funding is no longer 
attached to the process of training, then there are economic incentives to reduce the 
length of courses, prompting accusations of quality dilution. However, it could be 
argued that the ability to alter both the length and content of courses may lead to a more 
efficient use of scarce resources. 

• Creating cash flow problems 

ORF may lead to cash flow problems among training providers whose costs are incurred 
on an on-going basis but whose income is uncertain and ‘lumpy’; expenditure and 
income may not be well aligned over time. Estimating when trainees will achieve 
outcomes may not be easy, and the assessment and verification of outcomes may 
increase the uncertainty of funding triggers being reached. This problem will be even 
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greater for occupations commonly regarded as requiring ‘high cost’ training, such as 
engineering. The higher the level of ORF, the greater these problems may become. 

• Threatening/protecting some types of training provider 

Certain types of training provider may respond more quickly to the ORF regime than 
others. Providers whose main activity is the provision of training on a commercial basis 
may be quicker and less reluctant to rid themselves of occupational areas and types of 
trainees in which, and for whom, it is difficult and costly to trigger ORF payments. 
Other providers (such as employers and industry associations) may continue to provide 
such training, but only by cross-subsidisation from their other activities. Specialist 
providers, on the other hand, may suffer greater financial pressure and may, therefore, 
be forced to exit the market altogether. 

In addition, an increased emphasis on funding outputs may act as a barrier to entry for 
new providers. Unlike existing ones, they will not have a stream of outputs arising from 
trainees who have started earlier, so the cash flow problem for them is likely to be even 
more acute. This will raise barriers to entry and reduce levels of competition faced by 
existing training providers. Flexibility in the short-term might, therefore, reduce the 
breadth of provision in the long-term. 



2.4.3 ORF: reducing administrative costs and bureaucracy? 

It is often claimed that ORF can reduce administrative costs and bureaucracy. However, 
this is by no means certain, due to the implications of addressing some of the negative 
consequences of ORF through the ORF mechanism itself. For example, the ORF 
mechanism adopted may be complicated by the desire to make the system more 
equitable by recognising, for example, that not all trainees start with the same human 
capital endowments. As a result, the same output may trigger different payments 
depending on the background of trainees. This may require pre-training verification and 
therefore add to the administration and bureaucracy of the system. Similarly, rewarding 
part outputs or a ‘basket’ of outputs will add to the number of outputs that need to be 
monitored and rewarded. 
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2.4.4 ORF: enhancing transparency and accountability? 



• Appearing to add value 

A training programme creates value when, as a result of training, a participant’s 
prospects in the labour market exceed what they would have been in the absence of such 
an intervention, i.e., when the programme adds to the stock of human capital, as 
discussed above. However, observing a trainee’s prospects ‘before’ training and his/her 
prospects ‘after’ training does not necessarily indicate value-added. Other confounding 
variables (not always observable to the evaluator) also determine labour market (and 
qualification) prospects. These can range from situational factors such as access to 
transportation to attitudinal factors such as motivation and attitudes to training and/or 
work as well as the labour market context. Furthermore, the mix of participants in the 
training programme can influence how training programmes appear to fare. For 
example, a programme consisting of large numbers of high-potential participants will 
fare much better than one consisting largely of participants with low potential, 
irrespective of the quality of the training intervention. In other words, ORF contracts 
may encourage contractors to create the appearance of valuable training without actually 
producing real changes in the clients they serve, even when value-added measures of 
output are used, thereby distorting the transparency of the input/output relationship. 

• Failing to reward all aspects of value added 

On the other hand, output-related funding may not reward all the additions training 
makes to a person’s stock of human capital. This is especially the case where some 
progress has been made towards a particular performance goal, but the candidate has 
just fallen short of it. A good example here are systems which fund training providers 
on the basis of candidates who achieve a qualification. There may be cases where 
candidates are not able to complete the entire qualification for one reason or another, but 
who nonetheless attain part of the qualification. In addition, the value added through 
training may be multi-dimensional, which may not be captured in the definition of 
output. As a result, the ORF regime may not recognise all of the ways in which the 
stock of human capital can increase. 
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Bending the rules 



• 

Financial probity and reliability can be compromised by ORF unless certain checks and 
balances are instituted. A training organisation whose aim is the promotion of learning, 
and which searches out income for that purpose, or which has strong links with an 
industry and its traditions, may value its reputation sufficiently to overcome any 
temptation to ‘bend’ in order to secure output-related funds. However, other 
organisations might be more vulnerable to these pressures. For example, linking 
funding to qualification attainment where training providers are involved in the 
assessment process may lead to a conflict of interest where the training provider could 
lower the assessment criteria in order to secure output-related funds. Linking funding to 
job placements can also be open to abuse, with training providers providing trainees 
with fictitious employment in order to trigger payment. This danger is further enhanced 
in situations where providers have responsibility for all three parts of the process - that 
is, recruitment of participants, the training programme and job placements. In addition, 
incentives to bend the rules may be stronger where the proportion of funding dependent 
on reaching output triggers is high. If such potential exists then the principles of 
transparency and accountability may become blurred. 

2.5 Summary 

This section has highlighted the benefits of ORF. These can be summarised as 
providing an incentive to encourage: 

• more flexibility in the type of training provision offered 

• improvements in performance 

• increasing value for money by providing an incentive to fulfil certain achievements 
and discouraging ‘time-serving’ in training which do not lead to outcomes 

• simplifying administration and clarifying audit requirements 

Set against these benefits are a number of operational issues which need to be tackled 
before ORF can be introduced, as well as a number of consequences and implications 
which can arise from its implementation and use. These can be summarised in a series 
of policy questions: 
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• What level of ORF is required to reap the optimum benefits of tying funding to 
output? 

• Is ORF passed on to all those in the contracting chain? 

• How is output defined? 

• How is output measured? 

• How are funding uncertainties minimised? 

• Do the measures of performance used to financially reward and penalise training 
providers act as good proxies for the long-term labour market prospects of trainees? 

• Are only the best candidates ‘creamed off to output-related-funded programmes? 

• To what extent are eligibility criteria used to counter the danger of ‘creaming’ by 
requiring training providers to ‘dredge’? 

• How is the participation of disadvantaged members of the labour market protected? 

• What consequences does ORF have for the content and occupational spread of 
training provision? 

• Does ORF lead to cash flow problems among training providers? 

• Are particular types of training provider more at risk than others from the operation 
of ORF? 

• Does ORF raise barriers to entry for new training providers? 

• Are the administrative rules and regulations that accompany ORF simpler or more 
complicated than other systems? 

• Does ORF reward the value added by training intervention? 

• Does ORF reward all or only some of the value added by training providers? 

• What checks and balances are in place to ensure probity and reliability? 

• What mechanisms are used to counter these problems? 

• How successful is ORF in achieving its aims? 



Equipped with this broad overview, we now turn to a number of real world examples of 
the use of ORF within and beyond the EU. 
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During the course of this project, considerable effort was devoted to the identification of 
countries both within and outside of the EU which are using, or have had experience of, 
ORF for VET programmes. The result of these searches and enquiries confirmed our 
original suspicions that the US and the UK offered the best examples of its usage, with 
only piecemeal usage elsewhere in the EU. Indeed, the UK government, wishing to 
learn from international experience of ORF, commissioned a study in 1993 to review 
ORF in six countries - France, Germany, Sweden, Australia, Canada and the US (Green 
et al., 1993: 3) Tellingly, only the US was able to offer any pointers for the 
development of ORF. Our own enquiries for this study revealed much the same. . 

In the case of the US, ORF has been in use since the Job Training Partnership Act 
(JTPA) of 1982. This was designed to create an outcomes-driven training programme 
for the disadvantaged, which would demonstrate measurable results and do so without 
excessive costs. The UK’s use of ORF can be dated from the early 1990s and the 
establishment of Training and Enterprise Councils (TECs) to administer government- 
funded training programmes. The UK and the US, therefore, provide the two most 
obvious examples of ORF in practice, not least because there is a considerable 
descriptive and evaluative literature on its operation in both countries. This section of 
the Report will summarise this material in the light of the policy issues raised earlier. 
The striking feature of the review is the finding that the British emphasis on ORF as a 
tool of performance management is much stronger than that of the country in which the 
concept originated. In addition, this section will provide information on the issues 
facing those Member States - such as the Netherlands - which plan to introduce ORF in 
the near future; this section is divided accordingly. 
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3.1 ORF in the United Kingdom 



3.1.1 Rationale and Operation 

A network of 82 Training and Enterprise Councils (TECs) covering England and Wales 
was established in 1991. 1 They were set up as locally-based and employer-led 
institutions charged with contracting with government ‘to plan and deliver training and 
to promote and support the development of small business and self-employment within 
their area’ (Department of Employment, 1988: para 5.7). In 1998, the number of TECs 
had fallen to 78 due to two mergers, one declared bankruptcy, and one closure (Cobbold 
and Martin, 1997; Times Higher Educational Supplement, 21 November 1997). 

The majority of TECs’ funding comes from training programmes; around 5% comes 
from employers and individuals, about 1.5% from the EU, with the rest coming from 
central government (House of Commons, 1996: xxxvi). Three-quarters of this goes to 
fund training for young people entering the labour market for the first time (initial 
training) and for those seeking to re-enter the labour market after a period of 
unemployment (the dislocated). 

These schemes are Youth Training (YT), which caters for 16-17 year olds who have left 
school, and Training for Work (TfW) which caters for workers aged 18-63 years old 
who have been unemployed for six months or more. YT is aimed at providing young 
people with vocational qualifications at or above National Vocational Qualification 
(NVQ) level 2 2 , whereas TfW is designed to get the unemployed back into work and/or 
improve their vocational skills. Young people gain access to YT through the Youth 
Credit (YC) system, in which public funding is routed through the individual young 
person rather than the training provider in an attempt to empower the learner (see 
Felstead, 1993; Youthaid, 1997). However, the YT programme itself and the 
contractual relationships which surround it are the same. As a further complication, 

1 At the same time, 22 Local Enterprise Companies (LECs) covering Scotland were established. Since 
LECs have a much wider remit covering environmental, community and local economic development 
issues in addition to TEC-like responsibilities for the local implementation of national training and 
enterprise programmes, the discussion will focus on TECs. 

2 NVQs are awarded at five levels. These range from NVQ level 1 at the bottom to NVQ level 5 at the 
top. Government-funded training schemes, in the main, focus on providing training at NVQ level 2 and 
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young people can also enrol in a Modem Apprenticeship (MA) which takes as its basic 
aim the achievement of NVQ level 3. 

The function and institutional set up of TECs makes them hybrid organisations. While 
they are private organisations which deliver public services paid for by the taxpayer, 
they are different from a private commercial company working under contract to 
government - in this case the Department for Education and Employment (DfEE). 
While their formal responsibilities are the same as those of any other private company, 
they operate on a not-for-profit basis which prevents any surpluses generated from being 
dispersed to shareholders. Instead, surpluses can only be spent on activities considered 
important for the area, or else carried over into the next financial year (Jones, 1995). 

In addition to the principle that TECs must be locally-based and employer-led, there was 
a third condition - they must stand or fall on their own performance. This principle is 
the one most relevant for this Report. In the then government’s words, this meant 
‘attaining better value for money, greater efficiency and a higher return on investment’ 
(Training Agency, 1989b: 4). There are several inter-related ways in which the TEC 
system is designed to achieve this aim; the most visible is the annual publication of TEC 
performance league tables. The institutional and ORF framework within which TECs 
operate is set out at Figure 1 below: 

Figure 1: Institutional Framework of TEC Network 

Department for Education and Employment 
(DfEE) 



Contractual arrangements, training programmes agreed, 
output targets set and ORF regime. 

' 4 

Training and Enterprise Councils (TECs) || 

As above, although output targets may differ. 

^ 4 

Training Providers 
(Youth Training, Training for Work) 



NVQ level 3. Level 2 refers to basic entry qualifications for employment, while level 3 is pitched at 
craft/intermediate employment (see Felstead, 1997). 
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The delivery of government-funded training by TECs is measured according to six 
criteria (Performance Indicators and Analysis Unit, 1997). 

For Youth Training (YT) the criteria are: 

• cost per output point (each NVQ achieved carries output points) 

• number of output points per 100 leavers 

• number of qualifications gained per 100 leavers 

For Training for Work (TfW) the criteria are: 

• cost per output point (points here are awarded according to NVQ level and jobs or 
places on full-time education courses) 

• number of jobs acquired per 100 leavers 

• number of qualifications gained per 100 leavers 

Thus the output criteria for YT and TfW include the number of qualifications, the level 
of qualifications and job placement rates. They are compiled in the Inter-TEC 
Comparison Tables and are published every September. This allows the government to 
identify high- and low-achieving TECs. 

Increasing competition among training providers is designed to enhance efficiency gains 
- ORF being one of the means of doing so. Other means are also deployed. Every year 
TECs bid for funding based on the volume of training (measured by training weeks, on- 
programme payments or starts/enrolments), the number of qualifications and jobs 
(outputs) they expect to deliver, and the total cost of doing so. Separate bids are made 
for YT and TfW (as well as for enterprise programmes not discussed here) and are 
outlined in an annual Business Plan (Training Agency, 1989a: 25-26). The Business 
Plan quantifies the plan for both training programmes including: the number and type 
of trainees catered for; estimated costs; broad plans for the distribution of training by 
occupation; and the performance level the TEC expects to reach for each programme. 
The DfEE then negotiates with each TEC on the volumes, outputs and prices contained 
in the bid. Following this process, agreed volumes, outputs and a total contract price are 
written into the TEC’s Business Plan for the forthcoming year. 
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The total contract price is divided into two parts: payment for specified volumes of 
training and payment for specified outputs. The total contract price has a ceiling, as do 
volume and output payments. In other words, those who exceed their contractual 
requirements do not receive extra funding; however, those who fall short of their 
contractual obligations suffer financially. 

The TEC receives payment upon production of the relevant documentation. This can 
take a number of forms: attendance records are needed if funding is tied to the number 
of training weeks; a signed declaration confirming that an Individual Training Plan has 
been agreed is required if volume payments are made according to the number of 
programme starts; and written confirmation of trainee participation is needed if volume 
is measured according to the number of trainees on-programme on specified dates each 
month. Output payments, too, can only be triggered by the relevant documentary 
evidence. This includes written notification from the Awarding Body that a 
qualification has been awarded, an employer’s letter confirming that a former trainee 
has been in full-time work for at least 7 days within 13 weeks of leaving the programme 
or documentary evidence of enrolment on a full-time education course within 52 weeks 
of exit (DfEE, 1997c: 12-15 and 23). 

Payments for volumes are relatively simple - TECs supply DfEE with the relevant 
documentation and are paid the agreed rate. However, paying for outputs is more 
complicated as not all outputs are treated in the same way; certain outputs take longer 
and are more difficult to achieve than others. For example, under the 1997-1998 
arrangements, a YT trainee achieving an NVQ level 3 triggers two-and-a-half times the 
payment triggered by someone achieving level 2. Similarly, TfW trainees who get a job 
within 13 weeks of leaving the programme trigger output payments double those 
triggered by trainees who go onto full-time education or those who get NVQ level 2 
(DfEE, 1997d: 32 and 51). 

The funding arrangements also incorporate some allowances for those with Special 
Training Needs (STNs) who, by definition, require longer to achieve the same results as 
those in the ‘mainstream’. On YT those endorsed as having no realistic prospect of 
achieving NVQ level 2 are allowed to follow a programme leading towards level 1 
which, once awarded, carries a payment one-and-a-half times the payment a 
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‘mainstream’ trainee triggers on achieving NVQ level 2. Those with literacy and 
numeracy difficulties can follow specific non-NVQ courses designed to improve their 
basic skills in these areas. Once achieved, these trigger the same payments as triggered 
by YT ‘mainstreamers’ on the level 2 route. However, level 2 and above qualifications 
carry the same payments irrespective of trainee background (DfEE, 1997d: 32). 
Similarly, concessions are made for trainees with literacy and numeracy difficulties who 
enrol on TfW. However, no other allowances are made for those TfW trainees with 
STNs - an NVQ or a job carries the same payment (and number of points) whatever the 
trainee’s background (DfEE, 1997d: 51). 

As outlined above, the ORF funding system for TECs in the UK is a complicated one, 
so as to recognise that trainees may not all start with the same skills, and that 
achievement of a particular output for some represents more of an achievement than for 
others, i.e., it attempts to reward value-added. However, whether this is sufficient to 
take all the major aspects of value-added into account is another issue. This reveals that 
there is policy dilemma between a complex measure of output which is equitable, and 
one which is simple and less equitable but easier to administer. 

Detail apart, the balance between paying for training volume and paying for outputs has 
been shifting ever since the TECs were first set up. More and more emphasis has been 
given to payment by results. Initially, output-related funding contributed to 
approximately 10% of TEC budgets, but has since crept up to around 25% in 1992-1993 
and reached between 25-40% in 1993-1994 (Ball, 1993: 10). The shift in emphasis has 
continued. Today, ORE accounts for 25-35% of TECs’ YT budget and at least 75% of 
TfW funding (DfEE, 1997d: 22 and 49). The continued and increased emphasis on 
ORF of government-funded training has occasioned much debate and many evaluations 
of its consequences. It is to these that we now turn. 



3.1.2 Evaluation 

It is widely acknowledged that ORF can be a useful means of focusing TECs and their 
training providers on securing effective outcomes from taxpayers’ money. However, a 
number of studies have raised serious concerns about how the system has developed in 
practice (some of these are summarised in House of Commons, 1996: Part IV). 
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One such study was undertaken by Coopers and Lybrand (1995) who carried out an 
evaluation of a pilot funding scheme for Training for Work (TfW). This exercise 
involved an assessment of the consequences of changing the way in which TECs 
received their funding through, in particular, giving even more emphasis to ORF. The 
evaluation centred on seven TECs which agreed to pilot a new funding formula, 
whereby 25% of their training fees were paid per TfW trainee starter with the remaining 
75% paid according to each ‘successful’ completion of the programme (i.e. on trainees 
getting a job, becoming self-employed, entering full-time education or attaining a 
recognised qualification). This funding arrangement was almost a complete reversal of 
the mechanism in place at that time, where it was customary to pay 75% of the training 
fee for the number of training weeks completed with only 25% held back to reward 
‘successful’ completion (Employment Department, 1993). 

The evaluation was designed to assess the effect that these new arrangements had on the 
efficiency of the TfW programme. Two methods were used: an analysis of the 
statistical data relating to the seven pilot TECs and their comparators (selected on the 
basis of broad equivalence in terms of their economies, labour markets, size and 
performance); and a series of interviews with TEC officials, four or five training 
providers and up to two Employment Service offices in each area. 

According to the government’s interpretation of the evaluation, the results of the pilot 
funding scheme were very good, producing significant increases in both the number of 
jobs and qualifications achieved by trainees. The Employment Department said that 
‘positive outcomes per 100 leavers in the pilot TECs improved from 15 in 1992/93 to 40 
in 1994/95. Comparable figures for non-pilot TECs were 16 and 27. So from similar 
bases, pilots have improved by 25 percentage points compared to the 11 percentage 
points of the non-pilots’ (Employment Department, 1995b: 4). The evaluation report 
itself, however, was less sanguine about the success of this shift in funding emphasis. It 
suggested that most of the improvements were the result of factors, at best, only partially 
related or, at worst, wholly unrelated to the funding regime (Coopers and Lybrand, 
1995: 37-39). For example, pilot TECs were allowed to record ‘positive outcomes’ - 
that is, those getting jobs or going onto full-time education - and claim ORF payments at 
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any time during the 13 week period rather than on a particular spot date in week 13. As 
a consequence, the report concluded that: 

‘the improvement in recorded performance ... is primarily due to the 
earlier and more accurate reporting of the performance that is actually 
being achieved, whereas at non-pilots much of the performance goes 
unreported and unrecorded’ (ibid: 18; my emphasis). 

In other words, by placing a greater emphasis on the proportion of funding allocated 
according to output levels, there was a greater financial incentive to monitor outcomes 
more effectively. 

Coopers and Lybrand (1995) also found that placing more emphasis on ORF further 
shifts the focus of training activity toward meeting short-term labour market needs 
rather than equipping trainees with the skills necessary for long-term employability. 
This was revealed in a number of ways. The study found that in the pilot areas, 
increased attention was focused on the highest-performing courses - leading some to 
reduce or even close down training provision which did not quickly lead to jobs. In 
addition, a shift away from ‘high cost’ and/or long duration courses towards ‘low cost’ 
and/or short courses was detected. The evaluation also pointed out that the enhanced 
ORF regime made it more difficult for new providers to enter the market, especially in 
occupations commonly regarded as ‘high cost’ training areas. Signs were already 
evident that the provider network was shrinking, thereby reducing the range and choice 
of provision as well as the intensity of competition between providers. Finally, the 
study revealed that the pilot areas were more selective in their recruitment - for example, 
there was tendency for pilot TECs to recruit trainees from the more job-ready section of 
those eligible for the programme at the expense of those needing additional help, i.e. to 
‘cream’. (Coopers and Lybrand, 1995: 34, 52). 

Specific research into the consequences which ORF has for the training of those with 
Special Training Needs (STNs) has also been carried out (Meager, 1995). The data for 
this particular study come from two postal surveys. One focused on the entire TEC 
network and the other focused on a sample of 200 training providers, many of whom 
specialised in catering for the disadvantaged. Both had a response rate of around 65%, 
yielding information on 53 TECs and 126 training providers. The study was conducted 
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on behalf of a consortium of TECs and training providers with a particular interest in 
STNs. 



Several points of relevance to this Report emerge from the study. First, although a flat 
rate fee is paid to each TEC irrespective of the type of YT trainee or the training 
occupation (see above), TECs themselves can and do offer differential pricing to their 
YT providers (also see Felstead, 1994; Felstead et al., 1994). In 1995-1996, for 
example, two-thirds of TECs surveyed paid different volume prices to providers 
according to type of trainee and/or training programme. A third of these paid higher 
prices to training providers who took on trainees with STNs than to those catering for 
‘mainstreamers’ (Meager, 1995: 48; see also Employment Department, 1995a and 
DfEE, 1996). These types of volume contracts appear to be on the decline - they were 
offered by 73% of TECs in 1994-1995 compared to 67% a year later. A continuation of 
this trend is likely to make it increasingly difficult to offer YT to those with STNs since 
they are, by definition, a client group who are more difficult to train. The same 
principle applies to particular occupations - such as engineering - where it is more 
expensive to deliver training because of the cost of equipment, materials and trainers’ 
time. 

Secondly, TECs and providers were asked about the degree to which ORF is included in 
YT contracts, and the extent to which it varied between different types of provision. 
Here, there was much less variation. Around four out of five TECs offered no variation 
whatsoever in the payments they made to training providers for the outputs delivered 
whatever the type of trainee or occupational area. However, 15% of TECs reported 
varying payments according to trainee type, with another 6% varying payments 
according to occupational area (Meager, 1995: 51). These variations take the form of 
higher payments for some outcomes if trainees are designated as having STNs. Also, 
some TECs do pay more for qualifications in certain ‘high cost’ occupational areas - 
such as engineering (see Felstead, 1994). It is evident from this that TECs are under no 
obligation to pass on the ORF regime set by government to training providers. 

Respondents to the STN survey (Meager, 1995) were asked to identify factors which 
had an effect on the quality of training provision for those with STNs. Interestingly for 
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the purposes of this Report, ORF was frequently mentioned. Providers tended to see 
both ORF per se and the degree of ORF as having a negative impact on STN provision. 
For TECs it was not ORF per se that was problematic, but rather the degree of ORF (too 
high) which they felt - as did training providers - was having a negative impact on 
provision. However, not all providers were opposed to the principle of ORF, but some 
were concerned about the standardised nature of the outputs recognised for payment 
purposes. One voluntary sector training provider summarised these concerns as follows: 



‘We do not disagree in principle with ORF, and we do not disagree in 
principle with TEC quality assurance systems. The former can be used 
to focus objectives and the latter to measure them ... On TfW, however, 
we are unhappy that we only get paid for whole NVQs and not for units 
towards NVQs ... We would like to see other forms of measures 
introduced under TfW ... the main goal, as we see it, is that someone 
moves forward, makes progress. It is not always possible to get 
someone from unemployment into a state of employability through one 
TfW course, and the ORF regime tends to regard as failures, what may 
in reality represent considerable progress for the individual’ (quoted in 
Meager, 1995: 103, my emphasis). 



This reveals the potential inadequacies of an output measure which fails to record 
incremental achievements. However, as mentioned earlier, this has to be balanced 
against the disadvantages of using a more complex measure (such as reducing 
transparency and increasing bureaucracy). 



Other studies suggest that ORF may act to the detriment of other groups in society. For 
example, the present means of measuring TEC performance makes the launch of 
‘positive discrimination’ programmes financially unattractive to TECs (i.e. the provision 
of women-only training for occupations in which women are significantly under- 
represented). By their very nature non-traditional training courses are likely to yield 
fewer ‘positive outcomes’ per 100 leavers than sex-stereotyped courses, since sexual 
stereotyping continues to influence the job recruitment process (Curran, 1988; Collinson 
et al., 1990). In addition, it has been argued that the cost of single-sex training courses 
leading to employment in non-traditional occupations is likely to be high, as they are 
more likely to be classroom- rather than workplace-based (see Felstead 1995). 

Given these factors, only a few TECs (17%) in 1992 had set up courses for women in 
non-traditional areas (EOC, 1993: 31). As output performance indicators have assumed 
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a greater importance since that time, it is unlikely that the situation has improved. After 
all, positive action programmes carry the risk of putting downward pressure on a TEC’s 
performance rating, which may ultimately threaten its very survival (for a more detailed 
discussion see Felstead, 1995). 

The high level of ORF in the UK also heightens the likelihood that government funds 
may be misappropriated. With so much at stake, in terms of showing that training has 
produced outputs, there is a greater pressure on training providers to cut comers and 
even manufacture outcomes (qualifications or jobs). For example, in some occupational 
areas - such as business administration - NVQs can be awarded by several different 
bodies, some of which are considered less stringent than others when it comes to 
certification. Furthermore, as Stanton (1996) has pointed out, ORF places severe strain 
on the impartiality of the NVQ assessment process which sets out to attest to the ability 
of an individual to perform to workplace standards. This process relies heavily on the 
ability and willingness of those in a position to observe an individual’s workplace 
performance and make an impartial assessment of it. Often these assessors are the 
individual’s supervisor and/or trainer. If these assessors are in circumstances where 
their incomes (or the resources of the organisation for which they work) are affected by 
whether or not they judge a candidate to be competent, then their impartiality may be 
compromised. Some commentators have suggested that fraudulent claims may result: 

‘Output-related funding encourages people to look for loopholes but it 
wouldn’t be such a massive problem if you couldn’t fiddle the results. 

The only answer is wholly independent assessors who have no interest 
whatsoever in whether a candidate passes or fails’ (Charles Bell, 
spokesperson for Article 26, education human rights charity, quoted in 
Times Higher Educational Supplement, 14 March 1997). 

This suggestion has also been corroborated by a recent survey of 1,057 assessors which 
found that 38% felt that ‘many candidates pass who shouldn’t’ (Eraut et al., 1996: 65; 
Financial Times, 5 December 1996; Times Higher Educational Supplement, 29 
November 1996). 

Other fraudulent behaviour has also been identified. For example, a training provider 
with contracts with 13 TECs allegedly claimed training funds aimed at the long-term 
unemployed despite the fact that those on the course already had jobs. The scale of the 
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alleged fraud was so serious that it led to the closing down of one of the TECs 
concerned ( Financial Times, 25/26 October 1997 and 18 November 1997). 
Misappropriation of funds has also been uncovered with regard to qualifications 
attainment. Recently, a total of 25 TECs have been ordered to repay funds paid to them 
by DfEE for the qualifications generated by a Telford-based motor industry training 
provider ( Financial Times, 1 1 November 1997). There have been other cases of training 
providers registering non-existent trainees, double-counting candidates, inventing 
courses and placing trainees with ‘friendly’ employers for a short time in order to trigger 
output payments related to job placement ( People Management, 3 April 1997). 

The extent of these irregularities has been the subject of a recent investigation by the 
Committee of Public Accounts which estimated that £8.6 million in 1995-1996 was 
incorrectly paid by the government to training providers via the TECs (House, of 
Commons, 1997: vi). In response, the government has launched a fraud hotline as part 
of a package designed to clamp down on fictitious or inappropriate claims for funding 
from training providers ( People Management, 19 March 1998). 

The Committee of Public Accounts also used the opportunity to highlight one of the 
other dangers of ORF: 

‘We are concerned that output-related funding may encourage some 
providers of offer training which they find easiest and cheapest to 
deliver’ (House of Commons, 1997: xiv). 

For example, some occupational areas suffer from high staff turnover, and hence it is 
easier to trigger employment-based output payments by placing former trainees in jobs 
in these areas. A training provider for TECs made this point in evidence to the House of 
Commons Employment Committee during its investigations into the work of the TECs 
(House of Commons, 1996: xlii). She suggested that placing trainees in companies with 
a high turnover of trainees could make it easier for her company to meet its output 
requirements, rather than securing long-term employment for the trainees. Furthermore, 
the quality of the job - as measured by its content, pay and the nature of the employment 
contract - made no difference whatsoever to her ‘positive outcome’ payments. Where 
the emphasis is on getting trainees into jobs where the quality of that job is irrelevant (as 
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on the TfW programme), an economic incentive exists for training providers to seek the 
quickest, cheapest and easiest routes sufficient to trigger ORF payments. 

Both TECs and training providers are subject to the same pressures. TECs get paid the 
same amount, whether the output achieved is in a ‘low cost’ training area such as retail 
or a ‘high cost’ training area such as engineering and whether it is a relatively high- 
paying or low-paying job. In addition, the performance ratings reflect none of these 
differences. Furthermore, TECs are free to allocate any surpluses they generate 
(efficiency savings) to activities they wish to promote. There is, therefore, a financial 
incentive, a performance rating incentive and a surplus generation incentive to achieve 
outputs in sectors where qualifications or job placements are relatively easy to secure 
(e.g. clerical, hairdressing). 

An Employment Department study (1994) based on eight sample TECs in three regions 
provides evidence to this effect. For example, in one TEC area the majority of its 
outputs were generated from a national private sector organisation specialising in 
clerical/business administration training. At another two TECs, major contributions to 
output achievement came from national providers which specialised in clerical and 
hairdressing training. As a result, the training profiles of these TEC areas consisted of 
predominantly clerical and hairdressing provision (Employment Department, 1994: para 
19). Skewing provision towards these occupational areas, where training costs are 
lower, can enhance the TEC’s financial performance, its value-for-money rating, and its 
operating surplus (Jones, 1995 and 1996b) without regard to the long-term needs of the 
trainees or the labour market. 



3.1.3 Recent and Future Developments 

The principles of ORF can also be seen in other government-funded VET programmes 
in the UK. For example, there is an ORF component in the government’s New Deal 
(launched nationally in April 1998) for 18-24 year olds who have been unemployed for 
six months or more 3 . 



3 As a result, the eligibility criteria for TfW (now Work-Based Training for Adults) has been changed to 
cover the 25-63 age group, since 1 8-24 year olds will be covered by the New Deal. 
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Those eligible for the New Deal are offered four options: jobs for up to six months with 
one day a week off-the-job training leading to a recognised qualification; full-time 
education and training for up to 12 months, but only for those not qualified to NVQ 
level 2; placements lasting up to six months with the Environment Task Force 
comprising a minimum of one day a week (or equivalent) training; or work placements 
with the voluntary sector on the same terms as for those on the Environment Task Force 
(DfEE, 1997a). 

Interestingly, the provider payment process under each option contains an element of 
ORF. For example, full-time education and training providers receive 20% of their 
funding on ‘starts’, 50% on attendance, 20% on achievement of qualifications and 10% 
when trainees get a job expected to last more than three months (DfEE, 1997b: 7). At 
least 10% of provider funding under the other non-employer options (Environment Task 
Force and other voluntary sector organisations) is also tied to getting trainees into jobs. 
In addition, the funds paid to employers for the training component under the subsidised 
employment option are tied to outputs. A third of the £750 government subsidy 
available to employers is paid upon the production of an agreed training plan, another 
third is paid after the participant has attended for 13 weeks and the remaining third is 
paid at the end of the placement, provided the objectives initially identified are met. 

A key element, then, of the new government’s welfare-to-work programme - the New 
Deal - is funding tied to outputs. The principle of ORF is, therefore, firmly rooted in the 
UK. However, the government has recently announced its intention to review the role 
of TECs and the contracting and funding arrangements under which they operate 
(People Management, 28 May 1998). 
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3.2 ORF in the United States 



3.2.1 Rationale and Operation 

There are many similarities between the British TEC system and the US system of funding 
federal government training schemes (see Bailey, 1993; Bennett, 1994). However, the 
similarity of greatest significance for this Report is the emphasis the Job Training 
Partnership Act (JTPA) places on outputs as the means of securing accountability for the 
dispersal of public funds. Under the JTPA’s predecessor - Comprehensive Employment 
Training Act (CETA) - the US Department of Labor (DoL) exercised tight control over 
federally funded employment and training efforts. The DoL contracted with hundreds of 
non-profit or local government agencies charged with administering programmes at local 
level. They, in turn, subcontracted programme delivery to a network of training providers. 
Administrative control was exercised by regional DoL offices over all those involved in 
programme delivery. The funding emphasis, at this time, was on the numbers enrolled, the 
number of days trainees spent in class, certifications granted, and other input and process 
measures, since these were all assumed to lead to greater trainee competence, higher rates 
of employment and lower welfare demands. The central assumption governing the 1982 
JTPA reforms was that bureaucratic, process-orientated training policy had failed, and that 
an outcome-orientated system of accountability would lead to better results and greater 
efficiency. As one of its architects remarked: ‘the new system will be based on 
performance, not process’ (Senator Orrin Hatch, quoted in Donahue, 1989: 6). 

Under JTPA Private Industry Councils (PICs) 4 run local organisations - Service Delivery 
Areas (SDAs) - which administer training programmes in the area. There are around 650 
SDAs, one-third of which cover a population of close to 200,000 - the minimum threshold 
announced when SDAs were first set up. Many SDAs are contained within larger state 
agencies, which allows them to combine their relatively small resources with larger local 
and state organisations. SDAs, then, are local administrative units over which PICs have 
strategic control. PICs are, therefore, ultimately responsible for SDA performance. The 
institutional framework under which ORF operates within JTPA is set out in Figure 2 
below. 



4 A PIC comprises a majority of private sector members, a private sector chair, and a minority of 
representatives from education, labour, community groups and government agencies. Each PIC has 
responsibility for a particular SDA. 
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Figure 2: Institutional Framework of JTPA 



Federal Government 
(Department of Labor - DoL) 



Funding based on need. 
Performance Standards set. 
No ORF. 

i 

Individual States 



As above, but Performance Standards 
may be varied to reflect local conditions 
and small proportion of budget 
may be set aside for ORF payments. 

i 

Service Delivery Areas (SDAs) 
run by Private Industry Councils (PICs) 



ORF contracts tied to 
Performance Standards. 
Widespread use. 

i 

Training providers 



The US experience of performance standards, performance evaluation and ORF has lessons 
for European policy-makers who are considering introducing such a market-orientated or 
bottom-line approach to their VET delivery, or those who simply wish to learn more about 
ORF in practice. ORF has been used in the US for sixteen years, occasioning much debate 
and many evaluations of its consequences along the way. However, before considering 
these in detail, the Report will outline how the JTPA system operates. 

Various training initiatives (or Titles) are run under JTPA. These include: Title II-A 
programmes offering training to disadvantaged youth and adults; Title II-B schemes 
generally consisting of summer vacation training courses for disadvantaged young people 
(aged 14 or over); Title IH programmes aimed at displaced workers; and Title IV schemes 
providing specific training for Native Americans and migrant and seasonal workers, and 
intensive residential programmes mainly for high school dropouts (known as Job Corps). 





About a third of the total JTPA budget is spent on Title II-A and around a fifth is spent on 
each of the following: Title II-B, Title III and Job Corps (Jones, 1996a). 



Elements of performance management can be found in all of these programmes, but it is 
within Title II-A that systems of performance management have been developed furthest. It 
is also where most of the prominent evaluations in the field have been carried out (see 
below). Nevertheless, performance management is at the heart of the entire JTPA project 
and its associated institutions. It involves setting performance standards, evaluating 
programmes against these measures, and drawing up a series of rewards and sanctions to be 
triggered by various parties on the basis of their performance in relation to the measures. 
The emphasis on performance is the centrepiece of the JTPA legislation: 



‘The legislation must insist on performance. The current CETA system 
does not have any effective means of measuring program results or 
penalizing non-performance. The new legislation will provide 
standards for judging the programs for what they accomplish - by 
whether those trained are hired and earn more as a result of training. It 
will end federal involvement with the process of how people are 
trained. It will provide for measurement of the outcomes and remove 
the Federal government from involvement in the details of program 
operations’ (Senate Committee on Labor and Human Resources, 1992, 
quoted in Bailey, 1993: 10). 



The legislation characterises job training as ‘an investment in human capital’, and it directs 
the Secretary of Labor to develop ‘criteria for measuring the return on this investment’ and 
to translate these criteria into operational standards which SDAs must meet. Specific 
performance criteria for each type of programme are outlined in the legislation. 



Originally, performance criteria for adults under Title II-A were: 

• placement and retention in unsubsidised employment 

• increases in participant earnings 

• reduction in the numbers receiving welfare payments and the amounts paid out 



For youth they included: 

• attainment of ‘employment competitiveness’ as defined locally 

• completion of elementary, secondary or post-secondary education 

• enrolment in other training programmes, apprenticeships or the armed forces 
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Additional cost criteria were added to each of these targets. The legislation vested the 
Department of Labor (DoL) with powers to attach standards/targets to each of these aims 
and to provide optional models for the adjustment of national standards to take account of 
local SDA conditions. These are issued each year by the DoL. The standards themselves 
are set at levels intended to be reached by at least 75% of SDAs. Hence, some have 
claimed that the ‘performance standards are ridiculously easy to meet’ (Walker et al., 1986: 
26). In each SDA, state governors are responsible for which standards to adopt - those set 
nationally or those modified to take into account local conditions. In the early years, most 
state governors adopted the national standards; today, however, most evaluate SDA 
performance against locally modified yardsticks (Green et al., 1993). 

The distinctive feature of the US system is that, unlike the British system, performance 
management appears to be more unevenly adopted at different levels and entails the use of 
various kinds of rewards and sanctions (see Figure 2). Performance management at the 
state level is noticeably absent, with emphasis instead on procedural and regulatory 
requirements. JTPA funding allocations from the federal government are based on the 
criterion of need as measured by levels of unemployment and the numbers of disadvantaged 
individuals living in the state. Once states receive their funds, they must use the same 
funding allocation formula to disburse JTPA funds to SDAs (Svomy, 1996: 230). There 
are, however, additional regulations governing how the states spend their JTPA allocations. 
For example, under the original legislation 78% of the JTPA II-A allocation had to be spent 
directly on training provision, 40% had to go on youth provision, a maximum of 15% could 
be spent on administration and a further 6% could be used for technical assistance to SDAs 
or for incentive payments. 

Further down the chain the rewards and sanctions become less procedural and more 
financial in character. At the SDA level, both are at work. Here, the state monitors the 
performance of each of its SDAs against the performance criteria it has set for them (from a 
selection set by the DoL). If an SDA fails to meet these standards, the state is required to 
offer technical assistance to the troubled locality. If it misses the mark for two years 
running, the state governor and the DoL must impose a reorganisation plan. This could 
involve restructuring the SDA, ‘blacklisting’ offending contractors or assigning authority 
for the programme to a completely new entity. However, the imposition of reorganisation 
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plans is rare (Green et al., 1993). In addition to wielding such procedural sanctions over 
poor performers, states have the option of rewarding the better performing SDAs with 
bonus payments. The JTPA regulations allow 6% (now 5%) of a state’s allocation to be 
spent on special incentive payments of this kind. However, it is unclear how widespread 
this practice is. 

Nevertheless, the evaluations of JTPA (see below) suggest that SDAs are influenced by the 
performance targets which they are set and that they do respond to the rewards and 
sanctions to which they are subject. However, it is far from clear whether the procedural 
sanctions or the financial rewards are the more dominant force. Having said this, there are 
several factors which cast doubt on the strength of the financial incentive. Incentive 
payments make up only a small percentage of the state funds available for distribution, 
SDA funding levels remain protected irrespective of programme performance, and some 
states also use the incentive payment fund to provide extra help to those SDAs falling short 
of their performance targets. This is in marked contrast to the British system: local 
organisations - TECs - compete against one another for funds, their performance is 
monitored nationally with no allowance made for local economic variation, league tables 
against national standards are published annually and funding is tied to performance (under 
TfW at least three-quarters of funding is dependent on generating outputs). 

However, the nearer one gets to the delivery of the training, the greater the emphasis on 
ORF becomes. Yet, the JTPA itself does not stipulate how SDAs should deliver training, 
nor does it mention how outside training providers should be funded. These decisions are 
taken by each PIC for the local organisation - the SDA - under its control. Given the 
diversity among the 650 or so SDAs across the US and the considerable leeway they are 
permitted to exercise in this area, one might expect to observe a wide range of contrasting 
funding arrangements for training providers. The evidence, however, shows the contrary; 
they exhibit a marked preference for ORF-style contracts - tying part, or in some cases all, 
of their funding to the achievement of outputs. At the end of the nine-month launch period 
for JTPA (October 1983 to June 1984), 46% of all provider contracts were based on ORF 
principles, by the first full year of operation the proportion had risen to two-thirds, and by 
the end of the 1980s, 80% were using ORF-style contracts (Donahue, 1989; Hoachlander, 
1989). 
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Three reasons can be identified for such a preference. First, it is a means of ensuring 
provider accountability. Secondly, it ensures that training providers are working towards 
the same goals against which SDAs are themselves judged. Thirdly, it allows SDAs to 
disguise some of the administrative costs of running the programmes. JTPA legislation 
puts a limit on the proportion spent on administration (originally this was set at 15%, 
although since 1992 this has been raised to 20%). However, if an SDA concludes an ORF 
contract with training providers, all payments made under this arrangement count as 
training costs and none are counted as administration. This heightens the incentive to offer 
training providers ORF contracts, since many SDAs find it difficult to meet the 
administrative cost constraint. In addition, it encourages SDAs to transfer administrative 
functions - including important ones such as trainee recruitment and placement - on training 
providers. 

There is, however, little information on the specifics of the ORF contracts used by SDAs - 
indeed it is difficult to determine which types are the most common. Nevertheless, it has 
been suggested that a typical ORF contract is one in which some 10-20% of funding is 
contingent on certain outputs being achieved. Other, more stringent ORF contracts have 
been identified. For example, in Baltimore some of the early contracts paid 20% on 
enrolment, 40% on mid-point proficiency tests, 20% on end-of-programme tests and 20% 
on job placement at or above the minimum wage. More recently, the focus has shifted (in 
line with the changes in the DoL’s standards) towards end of programme and post- 
programme triggers. It now pays nothing on enrolment, 40% at the mid-point, 30% on 
completion and 30% on placement achieved within 30 days of completion (Green et al., 
1993: 21). Despite these examples, the evidence suggests that the US system of linking 
provider funds to performance is less extensive than currently practised in Britain. Indeed, 
at all levels in the US system the emphasis on ORF appears weaker. Nevertheless, 
evaluations of JTPA have paid particular attention to the practical consequences ORF has 
for training delivery in the US. 



3.2.2 Evaluation 

The eligibility criteria for participation in JTPA programmes have a major bearing on 
how ORF works in practice. Title II- A programmes are restricted to those with family 
income below a level denoting ‘relative poverty’ or those who receive some form of 
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public assistance (Devine and Heckman, 1996). Hence, there is no requirement that 
participants need be unemployed and there is no restriction on the degree or duration of 
economic disadvantage. According to some estimates (e.g. US General Accounting 
Office, 1989) there are between 10 and 39 million Americans eligible for JTPA, yet only 
about one million of these participate each year. How, then, are these people selected 
from the many who are eligible? 

In principle, procedural penalties for falling short of performance standards and 
financial incentives for exceeding them (as discussed above) discourage SDAs from 
serving the more difficult-to-train social groups, since a focus on them raises costs and 
makes it more difficult to meet performance standards. Awarding training providers 
with ORF contracts further encourages ‘creaming’ - that is, the non-random selection of 
participants. Both suggest that creaming is likely under JTPA and that ‘service 
providers choose program participants very carefully’ with job-readiness a key 
consideration (Walker, et al., 1986: 24). However, the form creaming takes and its 
significance is an empirical question which many JTPA evaluations seek to address. 

At a crude level, patterns of JTPA participation show that some groups are over- 
represented in the programme compared to their share among those eligible, while 
others are under-represented. For example, Donahue (1989: 15) found that women were 
under-represented by 7%, Hispanics by 9%, welfare recipients by 5% and high school 
dropouts by 35%. The over-represented groups included men (9%), whites (7%), blacks 
(30%) and high school graduates (16%). The evidence also suggests that the shift from 
a process-based system of funding under CETA to a performance-based system under 
JTPA has tilted the balance against the disadvantaged in favour of the more employable 
fraction of those eligible for JTPA entry. The groups under-represented in JTPA are the 
same groups which suffered a drop in representation following the switch from CETA 
to JTPA. The unemployed, women, minorities and high school dropouts all fell as a 
proportion of enrolled trainees; whites, the employed or recently employed, men and 
high school graduates saw their share rise (US General Accounting Office, 1985). 
Furthermore, screening may also take place within these groups. Training providers will 
often be able to observe - via interviews, application forms and references - 
characteristics which government training programmes may have difficulty altering, but 
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which are likely to affect labour market success. These include age, intelligence, family 
attitudes towards work and so on. 



These findings raise a major policy issue: is the pool of eligible participants so large that 
it allows providers to systematically identify the most promising candidates for training 
while weeding out those least likely to get jobs and/or stay the course (i.e. to make it 
look as though they are adding more value than they really are)? To make this particular 
point Donahue uses an analogy worth quoting at length: 



‘Harvard Business School graduates tend to earn more, over the course 
of their careers, than the average for the population as a whole. If the 
Harvard Business School selected its students randomly, this would be 
clear evidence that sitting through two years of case studies increased 
business acumen and earning power. But students are not selected 
randomly. The screening process is sufficiently rigorous that the people 
selected for admission would still tend to earn somewhat more than the 
population average if they spent two years in the Peace Corps, or 
learning to play the harp, instead of studying cases. It is impossible to 
distinguish conclusively between the value added by the professors and 
the value identified by the admissions office. A similar uncertainly 
complicates JTPA’s contractual system of accountability’ (1989: 17; his 
emphasis). 



This view also indicates the interconnected issues of screening (or creaming), value 
added and accountability. 

More sophisticated analyses of the extent of creaming have also been carried out. Like 
the studies reviewed above, these start out by comparing the social characteristics of 
JTPA participants with those eligible for, but not included in, the programme; their 
findings are similar. Anderson et al.’s (1993) examination of the Title II-A programme 
in the State of Tennessee, for example, found that women, the less educated (high 
school dropouts in particular), benefit recipients and older workers are significantly 
under-represented in JTPA programmes. However, bivariate comparisons may capture 
other confounding factors: for instance, dropouts are almost five times more likely to be 
benefit recipients with a physical or mental disability as well as being educationally 
disadvantaged. This shows the importance of undertaking multivariate analyses of the 
chances of being selected for inclusion in the JTPA programme. Using this approach, 
the better educated are significantly more likely to be selected to participate, but there is 
no additional dropout effect on the chances of selection. Similarly, gender and ethnicity 
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have no significant effect on the probability of selection, despite bivariate suggestions to 
the contrary. In other words, Anderson et al.’s (1993) analysis suggests that the only 
real selection criteria is better education; women, ethnic minorities and dropouts tend to 
be less educated rather than being directly discriminated against. 

For creaming to take place, one also needs to show that a biased selection of participants 
leads to higher placement rates than would occur had the trainee population exactly 
mirrored those eligible to take part. Adopting such an approach, Anderson et al. (1993) 
show that creaming does take place. Simulations suggest that placement rates in 
Tennessee would fall from 71% to 62% under these counterfactual circumstances. 
Although modest, placement rates would fall by nearly one-quarter if JTPA were 
focused exclusively on high school dropouts - a group which faces high and multiple 
barriers to employment. However, the multiplicity of barriers they face would - unless 
factored into the evaluation system - prompt creaming within the dropout population. 

Other studies have sought to examine the effect that different performance regimes have 
on the nature of the training delivered under JTPA and the extent of creaming. Most 
notably, Cragg (1997) assesses inter alia whether states which offer SDAs with 
relatively high rewards for exceeding performance targets and harsh sanctions for falling 
short have a greater tendency to cream off the more able individuals. The analysis is 
based on a nationally representative survey which allows one to pick out a sample of 
individuals eligible for JTPA (and its predecessor, CETA) as well as a sample of 
individuals enrolled on the programme. 

Using this approach Cragg examines the consequences of inter-state variations in 
performance regimes. These variations are captured in four ways: first, up to 6% (now 
5%) of a State’s JTPA budget may be used for incentive payments to SDAs - the higher 
the percentage set aside for this purpose, the greater the financial incentives for meeting 
performance standards set. Second, states may choose to pass on to SDAs all or some 
of the performance standards they are themselves set by the DoL - as more standards are 
required, higher incentives are generated. Third, the likelihood that SDAs will be 
punished for poor performance can be proxied by the number of standards which need 
to be met in order to avoid punishment. Fourth, states may be lenient towards SDAs in 
implementing the rewards and sanctions regime; this can be detected by whether or not 
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states permit SDAs to make special appeals for unexpected difficulties encountered - 
29% of States allow special appeals of this type. 

Multivariate analysis can then be used to compare the enrolment probabilities of 
individuals. In addition to variations in performance regimes, other variables - such as 
gender, age, education, ethnicity, labour market experience and local unemployment - 
are also entered. The results of these analyses indicate that high incentive regimes 
encourage enrolment of the more able individuals from the eligible pool. They also 
show that creaming has become more prevalent after the switch to JTPA with its 
emphasis on performance incentives. 

The types of training offered under JTPA reflect the emphasis placed on job placement 
rates. Four types of training are offered: classroom instruction; on-the-job training; help 
with job search; and work experience. 5 In general, the lengthier the training the lower 
the job placement rate. So, for example, work experience and classroom-based training 
take, on average, longer to complete, yet they have the poorest placement records. On 
the other hand, the least intensive JTPA type of training - job search - posts almost the 
highest placement rate. The emphasis JTPA places on getting trainees into jobs and the 
use of ORF provider contracts has seen a pronounced shift towards types of training best 
placed to boost placement rates. As a result, the change-over from CETA to JTPA saw 
job search enrolments rise by 535% and on-the-job training grow by 80%, while work 
experience fell by 84% and classroom instruction declined by 19% (Donahue, 1989). 
The implication is clear: the quicker and more reliably a type of training leads to 
placement, the greater the emphasis on that type of training. Such a shift in training 
provision may indicate that the system of performance management is creating more 
efficient types of training. However, what may be efficient in the short-term may be 
detrimental in the long-run. Furthermore, groups with the greatest chance of re- 
employment (e.g. whites, high school graduates) have seen their representation rise 
among the more successful types of JTPA delivery such as on-the-job training, while 
those with the poorest chances have seen their share fall. 



5 SDAs have considerable latitude in the kind of training provided in their areas. For example, Tennessee 
has 14 SDAs with a wide range of training approaches. In Nashville, on-the-job training is rarely offered 
to participants, but classroom training, work experience and job search assistance are quite common. In 
contrast, in the Memphis area, which has traditionally had a much higher unemployment rate, on-the-job 
training is the more common type of JTPA training offered (Anderson et al., 1993). 
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Some researchers have cast doubt on the relevance of the performance standards 
themselves, especially in the light of JTPA’s specified aim of delivering a measurable 
return on investment in terms of ‘the increased employment and earnings of participants 
and the reduction in welfare dependency’ (JTPA, section 106(a), quoted in Orr et al., 
1995: 5). Subsequent evaluations of the JTPA programme have suggested that the 
short-term performance measures used to reward or penalise SDAs and training 
providers have little association with participants’ longer-term employment, earnings 
and benefit dependency prospects. For example, Doolittle et al. (1993) calibrate these 
short-term performance measures against a longer-term evaluation of the performance of 
JTPA participants. The evidence for this study comes from the National JTPA Study 
which tracked the labour market performance of Title U-A applicants in 16 SDAs. 
Applicants were randomly assigned to two groups: two-thirds were allowed to enrol, 
while the remaining third were prevented from enrolling for 18 months (for a summary, 
see Bloom et al., 1997; for overview of evaluation techniques, see Fay, 1996). Doolittle 
et al. (1993) found that SDAs with strong performance according to the DoL’s short- 
term measures - such as the percentage entering employment on exit and average wage 
rates on placement - appeared to add little value to a participant’s prospects over a 30 
month post-training period. This has led some to argue that: 

‘performance measures are virtually useless in making rational decisions 
about effectiveness, even though they provide political protection 
because they make JTPA seem like a performance-driven program’ 

(Grubb, 1996: 26). 

However, it must be borne in mind that the US performance management system relies 
more on procedural ‘sticks’ than ORF ‘carrots’. 



3.2.3 Recent and Future Developments 

In recognition of some of these problems, the JTPA has been modified on several 
occasions. The 1988-1990 reforms, for example, were designed to shift the 
programme’s emphasis towards the provision of skills for sustained employment as 
opposed to job search activities whose effect might be more short-term. These reforms 
involved a staged transition from end-of-programme to post-programme performance 
measures. The new measures were devised to evaluate JTPA participant performance 




- 47 - 



50 



13 weeks after programme completion. These measures included employment rates, 
average weekly earnings, proportion of welfare recipients in work and average number 
of weeks worked. 

However, the most extensive changes to JTPA were the 1992 amendments to the 
legislation. The amendments had two aims. First, additional eligibility criteria were 
introduced in order to target the programme more directly towards those most in need. 
In addition to being economically disadvantaged, at least 65% of participants must have 
at least one ‘barrier to employment’ , such as having a disability, being a school dropout, 
lacking basic skills or having childcare responsibilities. In order to target youth, and 
especially those at greatest risk of economic exclusion, a separate Title (II-C) was 
created for youth programmes under JTPA. The 1992 amendments specified that a 
minimum of half of those recruited to this part of JTPA must be high school dropouts. 
In this way the legislators have attempted to counter the effects of creaming by imposing 
a ceiling beyond which participation would not be permitted. In other words, this is an 
example of ‘dredging’ on the basis of targeting policy to those most in need. Secondly, 
the 1992 amendments were a further attempt to shift the emphasis away from job search 
and short-term placement towards training and long-term employment. In order to 
encourage intensive training programmes, ‘stand alone’ job search activities were 
outlawed and instead greater emphasis was placed on skill acquisition, training and 
qualification attainment (Green etal., 1993). 
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3.3 ORF in the European Union 



Beyond the UK, it is difficult to find other EU examples of the use of ORF in the 
delivery of government-funded VET. However, experiments with ORF in higher 
education can be found in several Member States of the EU. In Sweden, for example, 
resources in higher education have since 1993 been allocated, in part, according to the 
outputs individual institutions generate; 60% of Swedish university funding is 
dependent on student grades, while the remaining 40% is allocated according to the 
number of full-time equivalent students taught at each institution. In other Member 
States the principle of ORF is also used to allocate higher education funds. In Finland, 
5% of university funding is tied to results, while in Denmark a proportion of funds is 
allocated according to the number of students who successfully complete their course. 
In Germany, some Lander have begun to experiment with the allocation of funds 
according to performance criteria. For example, in 1993 North-Rhine Westphalia 
decided to allocate 10% of institutions’ budgets according to the number of graduates 
over the previous three years, thereby giving institutions economic incentives to ensure 
that students do not drop out and that they graduate within the appropriate timescale. 
This represents a significant move away from the traditional German approach of 
funding entirely according to inputs (European Commission, 1996: 41-45). 

However, cases of ORF being used in VET delivery are rare. For some Member States, 
shifting the emphasis away from allocating block grants to institutions towards 
capitation funding represents a major move; yet this falls well short of ORF - although 
perhaps this represents an intermediate step towards considering the relationship 
between inputs and outputs. Since 1990, for example, vocational education in Denmark 
has been funded according to the so-called ‘taximeter’ system. The main principle here 
is that programmes are placed in certain price categories, and institutions delivering 
these programmes receive funding directly related to the number of full-time equivalent 
students enrolled at four points in the year (Hansen and Rasmussen, 1996). Funds to 
cover administration and other operating costs are also related to the number of students 
enrolled. In addition, institutions receive a basic grant dependent on the nature of the 
institution, its size and the number of courses offered (Gasskov, 1997). This still leaves 
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a large part of funding linked to the numbers enrolled, albeit counted four times rather 
than once a year. 

At least one EU country - the Netherlands - is contemplating introducing an ORE 
formula for its adult and vocational training centres. The new funding formula is 
expected to take effect from the year 2000. 6 The policy debate this has prompted 
revolves mainly around the mechanisms needed to ensure that institutional funding 
rewards value added. The method proposed differs significantly from the US tactic of 
tightening up on eligibility rules to minimise creaming and the British approach of the 
same outputs triggering differential returns in certain circumstances. 

The Dutch proposals attempt to tie institutional funding more directly to the difference 
between students’ educational level on entry and their level on exit. Ideally, this 
requires that all students be subject to a detailed test of their knowledge and skills upon 
entry and then again on exit. However, this would be cumbersome and costly to carry 
out. Instead, the proposals suggest that previous education at the time of entry be taken 
as the initial benchmark from which students start. This would, then, be used to 
determine an institution’s input-funding; that is, students starting from a low base would 
be funded for longer periods to follow courses of a prescribed length, while those with 
higher-level entry qualifications would be funded for shorter periods. Years which are 
repeated, or spent on a course which is later abandoned in favour of something else, add 
little to the student’s knowledge, and hence no additional funding would be allowed 
under these circumstances. 

The proposals also suggest an element of ORF, although the balance between this and 
input-funding is yet to be fully spelt out. What is clear, however, is that the award of an 
educational qualification is the output measure since: 

‘A diploma is the only type of educational qualification which is 
formally acknowledged by society at large, and people with diplomas 
have better employment prospects. It is therefore intended that output 
should in the future account for a greater proportion of total funding 
than it does at present’ (Ministry of Education, Culture and Science, 

1997: 13). 



6 Much of the subsequent discussion is taken from a document kindly provided by Peter Vrancken 
(Ministry of Education, Culture and Science, 1997). 
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The question of whether qualification outputs should be conceived more broadly to 
include modular certificates as well as completed diplomas has also been debated in 
Dutch policy circles. The proposals recommend a narrow focus on the latter since a 
broader focus - including modular certification - would bring many disadvantages. 
These include: an administrative burden as hundreds of thousands of modules are 
completed each year; the heightened risk of fraud; the cost of monitoring and 
maintaining effective supervision of the system; and increased likelihood that 
institutions would respond by awarding large numbers of low-level modular certificates 
rather than full diplomas in order to trigger ORF payments. 

Recently, agreement on these proposals has been reached. The proportion of funding 
comprising ORF has been set at 20% with 80% dependent on the number of students 
enrolled; bonus payments will be given for students with low entry levels; and courses 
will carry one of a small number of different prices. Interestingly, two elements of the 
proposed value-added arrangement will be excluded when the new funding regime is 
due to become operational in the year 2000. Students who enrol more than once for a 
course will still be counted for enrolment-based funding and the period of their 
enrolment will not be dependent on their entry level. These elements have been 
temporarily excluded from the system of value-added since the necessary data for their 
inclusion are not yet available. However, it has been agreed that these data will be 
collected and that these two elements will be subsequently included in the funding 
formula in (probably) 2004. 
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3.4 Summary 



From this review of case studies, several general points readily emerge: 

• First, beyond the UK it is difficult to find other EU examples of ORF in the delivery 
of government-funded VET. However, experiments with ORF in the educational 
system can be detected - in Sweden, Finland, Denmark and Germany. Nevertheless, 
some Member States - such as the Netherlands - are planning to introduce ORF into 
parts of their VET system in the near future. The longest standing use of ORF in the 
delivery of government-funded VET is to be found in the US. 

• Second, despite appearances to the contrary, the use of ORF in the UK and the US 
differs markedly. These differences include the emphasis it is given, its use along 
the contracting chain and the nature of the institutional environment in which it is 
embedded. 

• Third, ORF is not a simple and straightforward mechanism, easily understood in 
practice. On the contrary, it is often complex and varied. 

• Fourth, even within the UK and the US systems there are large gaps in our 
knowledge. For example, for neither country do we have sufficient information 
about the nature of training provider contracts and their consequences. 

However, the literature reviewed in the course of this section does provide us with 
concrete examples, evidence and well-grounded research material from which to reflect 
on the policy questions posed earlier. This opportunity is taken up in the concluding 
section which follows. 
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The principle of providing a financial incentive to reward a predetermined level of 
performance in VET delivery coupled with a financial penalty for those failing to 
achieve this level is deceptively straightforward. However, the use of ORF in VET 
delivery raises many policy issues worthy of note. Section Two has highlighted several 
of the more important to bear in mind when evaluating and assessing ORF regimes, 
while Section Three has provided us with the details necessary to answer these questions 
from a real-world perspective. In this concluding section of the Report we bring both of 
these sections together - reflecting on the policy questions posed by summarising some 
of the case study evidence presented. 

• What level of ORF is required to reap the optimum benefits of tying funding to 
output? 

The case studies indicate the range of emphasis that can be given to ORF - from 
75% in the case of TfW in the UK to around 5% in the case of JTPA in the US. 
Both claim to be set at the optimum level. 

• Is ORF passed on to all those in the contracting chain? 

This need not be the case. Indeed, while we know much about the contractual 
relationship between central government and its first-tier suppliers (TECs in the case 
of the UK and states in the case of the US), less is known about the contractual 
relationships further down the chain - especially those which govern the operations 
of training providers. 

• How is output defined? 

Typically, definitions include qualifications, jobs subsequently obtained, earning 
levels, enrolment on other programmes and reduction in welfare dependency. 

• How is output measured? 

On the whole, absolute measures of output are used more often. In other words, 
outputs are measured with little reference to the individual’s/institution’s starting 
position. Experiments with value-added measures are more exceptional. 
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* How are funding uncertainties minimised? 

In the US, the Federal government is insulated from funding uncertainty by basing 
state JTPA budgets on the basis of need - only 5% of their budget can be used for 
ORF. In the UK, TECs budgets are subject to a funding ceiling, with ORF operable 
only up to that point. 

« Do the measures of performance used to financially reward and penalise training 
providers act as good proxies for the long-term labour market prospects of 
trainees? 

Research evidence suggests that the proxies are not that good. However, it is not 
obvious what other proxies could be used, without putting training providers’ cash 
flow under even more pressure. 

* Are only the best candidates ‘creamed off’ onto output- related funded programmes? 
‘Creaming’ is an inherent problem to ORF. While its extent remains an empirical 
research question, its minimisation is a key policy challenge faced by all ORF 
systems. 

* To what extent are eligibility criteria used to counter the danger of ‘creaming’ by 
requiring training providers to ‘dredge’? 

Eligibility criteria have been used as a means to target programmes towards those 
most in need. This provides an important means of minimising ‘creaming’. 

* How is participation of disadvantaged members of the labour market protected? 

In some cases, a certain proportion of those enrolled must be from certain groups 
such as high school dropouts in the US. However, not all disadvantaged groups are 
protected in this way, and they may therefore lose out. 

* What consequences does ORF have for the content and occupational spread of 
training provision? 

In the absence of corrective measures, there tends to be a shift away from ‘high cost’ 
and/or long duration courses towards ‘low cost’ and/or short courses when increased 
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emphasis is given to ORF. Particular types of training delivery also suffer - such as 
classroom-based training - and training times are reduced. 

• Does ORF lead to cash flow problems for training providers? 

Reliance on funding triggered by outputs inevitably adds to training providers’ cash 
flow difficulties. As above, this may affect some training providers more than 
others. 

• Are particular types of training provider more at risk than others from the operation 
of ORF? 

UK evidence suggests that some training providers may have difficulty in remaining 
economically viable. While employers and industry associations may continue to 
provide ‘high cost’ training via cross-subsidisation from other areas, specialist 
stand-alone providers may suffer greater financial pressure and exit the market 
altogether. 

• Does ORF raise barriers to entry for new training providers? 

There is evidence that ORF has raised entry barriers to new training providers, 
especially those in ‘high cost’ training areas. This reduces the range and choice of 
provision as well as the intensity of competition between providers. 

• Are the administrative rules and regulations that accompany ORF simpler or more 
complicated than other systems? 

There is a policy dilemma between a complex measure of output which is equitable, 
and one which is simple and less equitable, but easier to administer. The ORF 
mechanism in the UK is complicated and fairly sophisticated, while both the US 
approach and the Dutch proposals are simpler and easier to administer. 

• Does ORF reward the value added by training intervention? 

In the UK, the same output carries a different level of reward depending on the type 
of trainee. However, differentiated payment regimes are complex and cumbersome, 
and they are not always passed on to all those in the contracting chain. 
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• Does ORF reward all or only some of the value added by training providers? 

Most ORF systems recognise only a handful of outputs. As a result, training 
providers are not always rewarded for the value they add to the stock of human 
capital. Recognition of part-outputs, however, makes the system more bureaucratic, 
as recently acknowledged by the Dutch proposals for ORF. 

• What checks and balances are in place to ensure probity and reliability? 

An effective ORF system requires that the output indicators not be fraudulently 
achieved by placing trainees with ‘friendly’ employers or by bending the rules in 
other ways. Systems which define output in terms of qualifications and rely on 
provider involvement in the certification process (such as those in the UK) are most 
at risk. 

• What mechanisms are used to counter these problems? 

US policy-makers have tended to counter many of these problems by revising 
primary legislation which governs the JTPA and, in particular, the eligibility criteria. 
UK policy-makers, on the other hand, have relied more on variations in the contract 
which governs its relationship with the TECs. 

• How successful is ORF in achieving its aims? 

Despite the evidence reviewed in this Report, it is difficult to be unequivocal on how 
successful ORF has been in achieving efficiency, reducing administrative costs, 
increasing flexibility, and enhancing transparency and accountability. Often ORF is 
but one element, albeit a significant one, in a host of institutional and funding 
reforms. In addition, ORF is rarely used as the sole instrument of performance 
management, rather it is one element within a package of measures. It is, therefore, 
difficult to isolate with certainty its effects. For example, both the UK and US case 
studies indicate that a number of reforms have been made partly as a result of 
introducing an ORF mechanism, e.g. narrowing the eligibility criteria for JTPA in 
the US. Whether these reforms were necessitated by the introduction of ORF and its 
associated imperfections or whether ORF has made the funding and training system 
more transparent (and therefore made existing problems more visible) is just one 
issue which is difficult to assess. 
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Introducing an element of ORF within the public sector funding of training requires 
careful thought in terms of both principle and practice. In principle, ORF may present 
many benefits to existing funding and training regimes. In practice, however, it does not 
operate in a vacuum. Hence, it may distort or enhance a number of other aims and 
policies. This Report has presented many of these abstract principles. It has then used 
the British system of funding government training schemes through TECs, the US 
system of JTPA delivery via the states and SDAs, and Dutch proposals to use some 
form of ORF to finance adult and vocational training to show how these principles relate 
to practical implementation and policy development. Many of these issues have wide 
applicability. Therefore, the discussion and cases presented provide material from 
which policy-makers and other interested parties may wish to borrow or learn. 
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Output-related funding in vocational education and 
training 

A discussion paper and case studies 4 



This report aims to provide information on the issues arising from 
and the practice of funding vocational education and training 
(VET) on the basis of programme outcomes rather than 
enrolment/attendance (i.e. switching the emphasis of funding from 
inputs to outputs). It outlines the policy issues which output-related 
funding (ORF) generates as well as a number of real-world 
examples of its use within and beyond the European Union (the 
Netherlands, the United Kingdom and the United States of 
America). The objective of the report, therefore, is to combine, 
within one document, discussion and evidence concerning ORF. 
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